Addition of time information

ABSTRACT

A method of adding time information to a data stream is provided. First and second timing sequences are linked with the data stream. The first timing sequence provides a permanent time record of when the data in the stream first originated. The second timing sequence is re-writable when the data stream is altered, for example by editing, so as to ensure an internally consistent time track is provided each time the data stream is changed. The timing sequences are derived from absolute references such as Global Positioning System (GPS) Time. The first and second timing sequences may be stored separately to the data stream. Two or more sets of second timing sequences may be provided, corresponding to different presentations of common sets of data streams.

FIELD OF THE INVENTION

This invention relates to the addition of time information to datastreams such as multimedia data streams, to facilitate editing andreplay of the data.

BACKGROUND TO THE INVENTION

Timestamps on multimedia data streams enable time information to beassociated with the data representing audio, video or other timedependent media. Recorded streams may require alteration as to rate ofplay, synchronisation with other streams or editing. The use oftimestamps embedded in the streams provides a marker for controllingsuch operations.

For example timestamps embedded in a data stream can be used to meterthe playback rate to ensure that the stream runs at the desired rate.Additionally if several streams are recorded simultaneously and requireediting, then operations such as cuts may be required at a particularrecorded time on each stream. The use of a timestamp ensures that thecuts can be precisely synchronised.

Timestamps are used within current video tape formats and DAT formats.Timing information in actual time (GMT) is typically laid down with theaudio and video streams as they are recorded. Subsequent editing of thestreams results in a time track with discontinuities where cuts havebeen made, and `false` timing data where inserts have occurred, so thatduring subsequent playback the timestamps do not bear a simplerelationship to the times at which the data should be played.

To avoid this a fresh time track may be laid down when editing, with thetime at which the edits were made. The original timestamps are thusoverwritten so that a new consecutive series of timestamps is available.This approach loses all information about when the original recordingswere made, so that additional tracks cannot be synchronised with theedited sequence, further editing will lead to cumulative errors, andrecordings cannot be indexed by when they were made.

Instead of timestamping with actual time (GMT), relative timestamps canbe used. These are relative times with independent origins and arelinked to every data segment (whether audio or video) by the hardwarethat captures the data. These timestamps often exhibit drift since theelectronic clocks in different hardware do not necessarily run withsufficient accuracy. When streams are edited, new sets of relativetimestamps are laid down so that the playout software does not have totake care of discontinuities where the edits have been made.

SUMMARY OF THE INVENTION

According to one aspect of the invention a method of adding timeinformation to a data stream comprises the steps of linking first andsecond timing sequences with the data stream, the first sequenceproviding a permanent time record of when the data first originated andthe second sequence being re-writable when the data stream is changed asduring editing, such that time information in the first sequenceprovides a permanent record of the originating time which is alwaysmaintained with any data preserved from the original stream, whilst thesecond sequence provides an internally consistent timetrack which isupdated each time the data stream is changed so as to be consistent withany derivative of the original data stream, for subsequent replay andediting purposes.

The timing sequences may be generated and transmitted with the datastream.

Where the data stream is to be edited "on line", the data stream andtiming sequences are preferably buffered before being transmitted toreproduction apparatus.

Where the data stream is to be edited from a recorded condition, thedata stream and timing sequences may be recorded in a synchronous orlinked manner and the timing sequence information may be embedded in thedata stream.

The first and second timing sequences may be embedded into the datastream together or the second timing sequence may be embedded into thedata stream after the first timing sequence. Alternatively the timingsequences may be stored separately from the data stream.

In the recorded condition, the second timing sequence may be re-recordedwhen, and if, the data stream is subsequently edited, so that aconsistent time sequence is ensured with no discontinuities, tofacilitate subsequent replay and editing of the data stream.

In a preferred embodiment of the invention the first permanent timingsequence is derived from an absolute reference such as, for example,Universal Coordinated Time(UTC) or Global Positioning System(GPS) Time.

UTC introduces leap seconds to remain synchronised to the rotation ofthe Earth. This complicates any software that deals with the smooth flowof data streams or calculates the times between data samples.Accordingly the use of GPS Time is preferred as this system avoids theintroduction of leap seconds and is easily related to UTC, following UTCto within 363 ns(95%) plus the accumulated leap second offset.

The second timing sequence is preferably in relative time units such asGPS seconds.

The timing information provided in the timing sequences may be providedby a series of timestamps.

The timestamps are preferably applied using a 64 bit time data format.

Each timestamp preferably comprises a 64 bit word. This 64 bit word maybe interpreted as a 2's complement binary representation in units of2×10⁻³² second. This permits negative times to be handled in the sameformat as positive times during calculations.

The series of timestamps may be positioned along the data stream withvarying spacing, or granularity. Thus for example the granularity can bealtered independently of the timestamp precision.

The first and second timing sequences may have different granularities.

The methods as aforesaid may be applied to a plurality of data streams,thus permitting linking of data streams, for example a separate videodata stream and audio data stream of common originating time with timingsequences applied according to the methods as aforesaid can be replayedin synchronism due to the identical first timing sequences applied tothe data streams.

This may be of particular use for synchronisation of live streams.

In accordance with a further aspect of the method of the invention,three or more timing sequences may be stored with the data. This permitsthe storage of two or more sets of second timing sequences, each ofwhich corresponds to a different presentation of common sets of datastreams. The first timing sequence will be common to all thepresentations. Where more than two second sequences are to be stored,some or all of the time sequences may be stored separately from the datastreams.

The invention also lies in apparatus for performing the aforesaidmethods.

The invention will now be described by way of example with reference tothe accompanying drawings in which:

FIG. 1 shows schematic views of apparatus which uses timing sequences,FIGS. 1a) and b) being schematic views of apparatus which applies timingsequences to live data streams and to data streams as recorded, and FIG.1c) is a schematic view of apparatus for editing data streams;

FIG. 2 is an example of time gating applied to data streams; and

FIG. 3 shows a schematic view of timing sequences applied using an ATM(Asynchronous Transfer Mode) network.

DESCRIPTION OF EMBODIMENT

FIG. 1 shows schematic views of timing sequences comprising a series oftimestamps being applied and used in connection with data streams.

FIG. 1a) shows the application of timestamps to live data streams inaccordance with the invention. A video camera 10 is used to convert anoptical scene into a live video data stream which can be recorded inknown manner for subsequent display as on a screen 12. A microphone 14converts sound waves into a "sound" data stream which can also berecorded in known manner and replayed via a loudspeaker 16. As the datais transmitted from the camera 10 and the microphone 14, first andsecond timing sequences are linked to the data streams.

The first timing sequence is an absolute time system and contains sourcetimestamps. These timestamps are set to the time at which data iscaptured and are synchronised to Global Positioning System (GPS) time soas to be compatible with international timing systems such as UniversalCoordinated Time (UTC). The source timestamps can be regarded as part ofthe data, just as much as the data describing the pixels making up theimage or the waveform samples making up the audio signal. The sourcetimestamps are permanent and not changed by editing or filingoperations.

The second timing sequence contains playback timestamps. These definerelative times at which data is to be replayed. The playback timestampsare not related to any particular time origin but are in standard timeunits of GPS seconds. The playback timestamps indicate the times atwhich the data in a recorded stream is to be replayed, thus for exampleallowing alteration in the play back speed of clips and insertion ofpauses. To this end, apparatus reproducing the data streams aspicture(s) and/or sound are controlled by the playback timestamps.

The data streams typically comprise a series of data segments. Thesource timestamp can be embedded into the data segments initially withthe playback timestamp embedded later, for example when the streams arerecorded onto a data store.

Alternatively during the initial recording of the live data streams, thesource and playback timestamps embedded together and with the samevalues made identical, so that if replayed the data will be played backat the same rate and in the same order in which it was recorded.

In FIG. 1a) the data players, for instance displays 12 and loudspeakers16, hold the live data with playback timestamps SP until real time TRreaches SP=TOff, where TOff is set identically for all players accordingto the maximum delay expected for any link, or adaptively by observinghow close any player's buffers have come to emptying. This permitssynchronisation of live sound and video data streams, for example whendata is transmitted over a satellite link.

FIG. 1b) shows a schematic view of time sequences being recorded ontodata streams for storage. A video camera 18 and microphone 20 transmitthe data stream and the data processing apparatus 22 applies thetimestamps. The spacing of individual timestamps in a time sequence,i.e. the granularity, can be varied independently of the timestampprecision. Times for data between timestamps may be deduced fromknowledge of sampling rates. In a system where one or several datasamples are passed as a single message, timestamps can be incorporatedin all messages, some messages, or in special messages interleaved withthe data.

The timestamps may be stored separately from the data stream. Byreferencing each timestamp to a byte in the corresponding data stream towhich they refer, it is possible to use timestamps with olderincompatible data storage hardware. At playback, the required video byteis selected and the timestamps supplied from a timestamp index by use ofthe video byte reference number.

The source and playback timestamps may be kept together, or can beintroduced at different granularities according to the application.

The timestamps are observed somewhere in the data processing apparatus22, and the data flow controlled accordingly. The data streams arestored with the applied time sequences on a storage means 24 and playedout later as described, for example, in connection with FIG. 2.

For replay of stored streams, the replay device computes a time offsetfrom the current real time TR be used by the players. For instance, ifthe streams to be played all start with timestamp S, and are to beplayed with all playback timestamps synchronised, and the time that thedisplay is to start is T, then the data with playback timestamp SPshould be held at the display until TR reaches SP+TOff where TOff=T-S.The offset thus allows users to skip backwards or forwards in arecording.

FIG. 1c) shows a schematic view of editing the data streams. The dataincluding source and playback timestamps are recorded on the storagemedia 26 with no special attention given to the timestamps, as forexample shown in FIG. 1b. Editing of the data stream occurs at theediting suite 28.

During editing the data stream is given a new set of playback timestampsthat define the relative times at which each timestamped section of datais to be played. The cut and edited data stream is joined to give anedited data stream 30. The original first timing sequence with thesource timestamps remains, with jumps in the track where data has beenedited.

The second timing sequence containing the playback timestamps isre-written by, for example, deriving a signal from the source timestampsvia offsets. The re-written second timing sequence provides a consistenttime track for playback purposes without any jumps in it. This removescumulative errors where the precise times of cut points differ becauseof incompatible sampling rates.

If several video clips are to be spliced together and an accompanyingaudio track is to be added, the source timestamps can be used by acomputer system to determine which parts of the video have associatedaudio tracks. The computer edits together the exactly correspondingsounds to any given video clip without the need for manual alignmentafter each cut.

During editing of a collection of streams that are to be synchronised,the playback timestamps may be set on a common basis so that only asingle time offset needs to be computed as in the case of uneditedstreams. This is performed by a controlling system interacting with timegating modules as shown in FIG. 2. Alternatively the synchronisedplayback can be initiated by giving an offset to be maintained betweenthe playback timestamps of each stream and real time. The offsets can beenforced in the destination machine such as a video display orloudspeaker driver, but in simpler systems the offsets can be enforcedas data is released from playback devices.

In FIG. 2 stored data streams 32,34 are fed into time gates 36,38. Thetime gates 36,38 initialise with the largest possible time offset TOffso that no data will be played out. The time gates 36,38 read the inputstreams until the first timestamp is encountered. This is held (as theplayout time will be a long way in the future), and its value isavailable as an attribute of the gate module.

A controlling system 40 reads the current timestamp values as suppliedby the time gates 36,38 from all the streams that are to besynchronised, and from these values determines at what time offset toplay the media stream to loudspeakers 44,46.

As an example of how offsets can be used, the initial playbacktimestamps may be 941 and 946, and known to be relative to the sametimebase. The controlling system 40 reads the actual time from the gatemodules 36,38, or some other source, and this may be 1000. Thecontrolling system 40 subtracts the current timestamps from the currenttime and adds a time quantity, for example 1 unit, to ensure that everypiece of data in the two streams is seen. The playback offset for eachtimegate is set according to the earliest playback timestamp (941), andso is set to

    1000-941+1=60.

This offset causes stream 32 to start at time 941+60=1001, and stream 34to start at 946+60=1006. At time 1010 both streams are running.

If the streams 32,34 are to be started simultaneously, then the later ofthe start playback timestamps is taken and the playback offset set to1000-946+1=55.

The controlling system 40 identifies that stream 32 is late, andforwards the stream, discarding data, until the playback timestamp ofstream 32 reaches 946 at real time 1001, when the stream will startplaying. Stream 34 also starts at time 1001.

Note that in these examples of offset use, the streams are synchronisedtogether. The choice has been whether to start the streams in a raggedfashion showing all data, or to jump forward to a time when all streamshave data.

The timestamps are applied in a time data format having a 64 bitrepresentation. Each timestamp is a 64 bit word interpreted as a 2'scomplement binary representation in units of 2×10⁻³² second since GPSepoch(Midnight, Jan. 5, 1980). The top 32 bits represent binary GPSseconds, the bottom 32 bits binary fractions of each second. Thus lowprecision code uses just the top 32 bits as a 2's complement number ofseconds since the GPS epoch. High precision code down to the nanosecondlevel can be accommodated by the bottom 32 bits, the time resolution ofthe system being about 0.2 ns.

With 2's complement the top bit of the 64 bit word represents whetherthe number is positive or negative. If the top digit is 0 the number ispositive, if the top digit is 1 the number is negative. By specifying2's complement rather than unsigned, and thus giving each timestamp apositive or negative value, library code can handle negative times, ietimes before GPS epoch, in calculations. This time format is only validfor times between the years 1915 and 2045, i.e. until the positive timeis reached at which the top bit is 1. This is sufficient for mostmultimedia timestamping requirements. Alternatively the rollover of thecounting at year 2045 can be taken into account.

The use of 2's complement in the system ensures that negative numberscan be processed easily. This reduces the chance of user error whenmanipulating different data streams using the addition or subtraction oftimestamps.

The retention of the original source timestamps allows data to beretrieved according to the times of recording (e.g. between 6 and 7 pmon 5th November), enables streams to be synchronised even if they havebeen recorded on different media, and makes it possible for a program tosystematically add new audio and video tracks to an edited sequence,even where it has pauses, speed changes or other effects. The sourcetimestamps also provide an unchanging reference time scale from whichplayback timestamps can be computed with no cumulative error aftermultiple edit.

The use of the GPS time standard makes high accuracy time hardwareavailable worldwide, avoids problems with leap seconds, and is easilyrelated to UTC. Discussion of GPS standards and associated accurancyfigures when related to UTC are contained in `Technical Characteristicsof the Navstar GPS, NATO Navstar GPS Support Group, June 1991.`

FIG. 3 shows a schematic diagram of how timing sequences are applied byuse of an ATM (Asynchronous Transfer Mode) computer network. The ATMNetwork 48 is a computer network which continuously sends small cells ofinformation between individual processing sections and so controls theinteraction of different sections such as a data collection section 50,a data storage facility 62 and a user display section 70. Switches 58,60are provided within the network 48 to direct the cells. Each cellcontains 53 bits of information with the network address of each cellbeing contained within a header 5 bits long. Each cell address istranslated within the switches 58,60 and due to the small size of theheader, the switches 58,60 process each cell quickly and direct themaccording to the network address. A network time protocol NTP is used tomeasure how fast the cells travel throughout the ATM Network 48. Thisallows for synchronisation between each section attached via the ATMNetwork 48.

Network time Protocols are discussed in, for example, Network TimeProtocol (Version 2) by D.L. Mills in DARPA Network Working Group ReportRFC-1119, University of Delaware, September 1989.

The data collection section 50 comprises a video camera 52, an Analogueto Digital(A/D) converter 54 and a processing unit and network interface56. The video camera 52 converts an optical scene into a live videoanalogue data stream. This analogue signal is sent to the A/D convertor54 for conversion to a digital signal before passing to the processingunit 56 where the first and second timing sequences are applied to thedata stream. The processing unit and network interface 56 is typicallyprovided by an Olivetti Research Laboratories ATMos board with a localdigital clock signal. The software protocol defining the addition of thetiming sequences containg the timestamps to the data stream is providedfrom the ATM Network 48.

The data storage facility 62 comprises a disc storage facility 64sometimes known as a disc brick, a signal convertor 66 and a dataprocessing unit 68. The data processing and interface unit 68 istypically another ATMos board and allows connection via switch 60 intothe controlling ATM Network 48.

The user display section 70 comprises a video display unit 72, a D/Aconvertor 74 and a data processing and interface unit 76. The dataprocessing unit and network interface 76 is similarly an ATMos boardconnected into switch 60.

A GPS receiver 78 receiving time pulses from GPS transmitters in orbitaround the earth is used to provide the absolute time reference for thetiming sequences. The receiver 78 is connected into the ATM Network 48by a further ATMos board 80, typically by use of an RS 232 interface,and the switch 60. The time pulse from the receiver 78 is used tocalibrate and synchronise the clocks on each of the ATMos boards.Typically the boards can be calibrated to within 100 μs although betteraccuracy can be achieved if necessary.

In use a video data stream from camera 52 has playback and sourcetimestamps applied at ATMos board 56. An information cell comprising oneor more timestamps is sent via switch 58 to the appropriate part of theprocessing network. Switch 58 allows combination of data streams fromthe data collection section 50 with those of the data storage facility62. Data streams from the data storage facility 62 may be stored withtimestamps or alternatively timestamps, in particular playbacktimestamps, can be applied to the data stream by ATMos board 68.

The different data streams are combined according to the playbackinstructions contained within ATMos board 76 and sent via the D/Aconverter 74 to the video display unit 72. The different data streamssupplied through the ATM network 48 to the video display unit 72 can besynchronised or offset relative to each other depending on the softwareprotocol supplied to the ATMos board 76 by the ATM network 48.

I claim:
 1. A method of adding time information to a data stream themethod comprising the steps of recording the data stream onto a storagemedium, linking first and second timing sequences with the recorded datastream, the first sequence providing a permanent time record of when thedata first originated; updating the second sequence when the data streamis subsequently edited, such that time information in the first sequenceprovides a permanent record of the originating time which is alwaysmaintained with any data preserved from the original stream, whilst thesecond sequence provides an internally consistent timetrack for theedited data stream, which timetrack therefore does not containsequential discontinuities corresponding to those which may have beenintroduced into the first sequence by the editing of the data streamwherein the timetrack is updated each time the data stream is changed,to facilitate subsequent replay and further editing of the data stream.2. The method of adding time information to a data stream according toclaim 1, wherein the timing sequences are generated and transmitted withthe data stream.
 3. A method of adding time information to a data streamaccording to claim 1, wherein the data stream and timing sequences arebuffered before being transmitted to reproduction apparatus.
 4. A methodof adding time information to a data stream according to claim 1,wherein the data stream and timing sequences are recorded in a linkedmanner and the timing sequence information is embedded in the datastream.
 5. A method of adding time information to a data streamaccording to claim 4, wherein the first and second timing sequences areembedded into the data stream together.
 6. A method of adding timeinformation to a data stream according to claim 4, wherein the secondtiming sequence is embedded into the data stream after the first timingsequence.
 7. A method of adding time information to a data streamaccording to claim 1, wherein the timing sequences are stored separatelyfrom the data.
 8. A method of adding time information to a data streamaccording to claim 1, wherein the first permanent timing sequence isderived from an absolute reference.
 9. A method of adding timeinformation to a data stream according to claim 1, wherein the secondtiming sequence is in relative time units.
 10. A method of adding timeinformation to a data stream according to claim 1, wherein the timinginformation in the timing sequences is provided by a series oftimestamps.
 11. A method of adding time information to a data stream,according to claim 10, wherein the timestamps are applied using a 64 bittime data format.
 12. A method of adding time information to a datastream according to claim 11, wherein the 64 bit time data format isinterpreted as a 2's complement binary representation so as to allownegative times to be handled in the same format as positive times duringcalculations.
 13. A method of adding time information to a data streamaccording to claim 10 wherein the series of timestamps is positionedalong the data stream with varying spacing.
 14. A method of adding timeinformation to a data stream according to claim 13, wherein the firstand second timing sequences have differently spaced timestamps.
 15. Amethod of adding time information to a data stream according to claim 1wherein the time information is added to a plurality of data streams, soas to allow linking of different data streams.
 16. A method of addingtime information to a data stream according to claim 1, wherein thetiming sequences are used to synchronise live streams.
 17. A method ofadding time information to a data stream according to claim 1, whereinat least three timing sequences are stored with the data so as to allowthe storage of at least two sets of second timing sequences, each ofwhich corresponds to a different presentation of common sets of datastreams.
 18. Apparatus for performing any of the methods according toclaim 1.